Display control apparatus and method for same

ABSTRACT

A display control apparatus operable to display an operation of a robot comprising: an obtaining unit configured to obtain measurement information of a processing target that includes at least one of the robot and a target that the robot operates; an emphasizing unit configured to generate emphasized display information that emphasizes a feature of a portion of the processing target, based on the measurement information; a composition unit configured to generate composite display information in which the emphasized display information and non-emphasized display information other than the emphasized display information are composed; and an output unit configured to output the composite display information to a display unit.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to a display control technique thatdisplays an operation of a robot.

2. Description of the Related Art

It is known that it is useful for understanding a situation when, inoperation teaching of a robot, movement of a target object, such as apart, or an inserting operation, or the like, is taught, if amisalignment of its position or orientation or, and force generated at atime of insertion of a target is visualized for an teacher. In the first“Next-generation Robot Intelligence Technology Development Project”(post evaluation) subcommittee, document 5-1, a user interface that aidsan teacher by superimposing, with a graph and a size of arrows for eachXYZ axis of a three-dimensional space, information such as that of aposition/orientation misalignment with the target or an acting forceonto an image of a camera attached to an end effector of a robot isproposed.

However, for a method of visualizing information by arrows or a graph,for example, in a case in which all of position misalignment,orientation misalignment, and a force acted are visualized at once,displayed content of an image becomes very complicated due to many arrowor graphs, and may impede an teacher's intuitive understanding.

SUMMARY OF THE INVENTION

The present invention is conceived in view of the above-describedproblem, and provides a display technique capable of improvingvisibility of a state of a processing target.

A display control apparatus operable to display an operation of a robot,the apparatus comprising: an obtaining unit configured to obtainmeasurement information of a processing target that includes at leastone of the robot and a target that the robot operates; an emphasizingunit configured to generate emphasized display information thatemphasizes a feature of a portion of the processing target, based on themeasurement information; a composition unit configured to generatecomposite display information in which the emphasized displayinformation and non-emphasized display information other than theemphasized display information are composed; and an output unitconfigured to output the composite display information to a displayunit.

By virtue of the present invention, it is possible to provide a displaytechnique that can improve visibility of a state of a processing target.

Further features of the present invention will become apparent from thefollowing description of exemplary embodiments (with reference to theattached drawings).

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a view for illustrating a system configuration a firstembodiment.

FIG. 2 is a flowchart for describing an operation of a first embodiment.

FIG. 3 is a view for illustrating a captured image of the firstembodiment.

FIG. 4 is a view for illustrating positions and orientations of eachunit of the first embodiment.

FIG. 5 is a view for illustrating a position misalignment image of thefirst embodiment.

FIG. 6 is a view for illustrating an emphasized image of the firstembodiment.

FIG. 7 is a view for illustrating an emphasized region of the firstembodiment.

FIG. 8 is a view for illustrating a composite image of the firstembodiment.

FIG. 9 is a view for illustrating a matching region of the firstembodiment.

FIG. 10 is a view for illustrating a matching image of the firstembodiment.

FIG. 11 is a view for illustrating a system configuration a secondembodiment.

FIG. 12 is a flowchart for describing an operation of a secondembodiment.

FIG. 13 is a view for illustrating a virtually constructed robotenvironment of a second embodiment.

FIG. 14 is a view for illustrating a system configuration a thirdembodiment.

FIG. 15 is a flowchart for describing an operation of the thirdembodiment.

FIGS. 16A and 16B are views of another embodiment for illustratingstates before and after emphasizing by causing a target part in acaptured image to be deformed by an action force generated by a contact.

FIGS. 17A and 17B are views for illustrating states before and afteremphasizing angle misalignment of a target part with respect to a fixedpart, according to another embodiment.

FIGS. 18A and 18B are views for illustrating states before and afteremphasizing a speed of a target part and a finger, according to anotherembodiment.

FIGS. 19A and 19B are views, of another embodiment, for illustratingstates before and after matching is achieved with an emphasized targetpart by translating and rotating the three-dimensional shape of thefinger.

DESCRIPTION OF THE EMBODIMENTS

Embodiments of the present invention will now be described hereinafterin detail, using the drawings.

First Embodiment

Below, explanation will be given of a robot information display system(a display control apparatus) that uses a captured image from an imagingdevice (a camera) mounted on a leading end of the robot (for example, anarm-type robot) to visualize (display control) having emphasizedposition information necessary when teaching.

In a conventional system, information such as position misalignment orangle misalignment in each axial direction of an operation target issuperimposed, with a size of an arrow or a graph, onto a captured imageof a camera. However, if many pieces of information are superimposed,there is a problem in that screen content becomes complicated, and it ishard to see minute misalignments.

Accordingly, in the present embodiment, visualizing is performed afteremphasizing a displacement and a size of a robot or a portion of atarget that the robot operates in a captured image, and output to aviewer is performed. With this, when teaching the robot, the viewer, byobserving a composite image that uses the image captured by the camera,can confirm position misalignment of a target object intuitively andeasily.

FIG. 1 illustrates an overall configuration of a robot informationdisplay system that emphasizes minute position misalignment in acaptured image of a part that a robot grips and inserts; and thatoutputs to a viewer.

A robot 101 is arranged so as to use a finger 102 attached to a leadingend to grip a target part 103, and perform an inserting operationtowards a fixed part 104 that is fixed so as not to move. A camera 105is arranged to the rear of the finger 102 on the robot 101, andfunctions as an imaging unit that captures a state of the finger 102,the target part 103 that the finger 102 grips, and the fixed part 104,and outputs a captured image. A tracker 106 measures, with the tracker106 itself as a basis position, a position and orientation of a targetcoil 107 attached to a leading end of the robot 101.

An image input unit 108 is connected to the camera 105, obtains thecaptured image captured by the camera 105, and holds it as an inputimage. A measurement information obtaining unit 109 obtains and holds,as measurement information, position and orientation information thatindicates a current position and orientation of the target coil 107obtained by the tracker 106. In addition, the measurement informationobtaining unit 109 holds environment information relating to the robot101 and an environment surrounding it, such as a position of the fixedpart 104 or a positional relationship, measured in advance, between thefinger 102 and the target coil 107 of the robot 101.

From the captured image and the measurement information and theenvironment information that the measurement information obtaining unit109 holds, a display information conversion unit 110 generates aposition misalignment image as display information, by processing thecaptured image. The position misalignment image adds positioninformation of the fixed part 104 and the target part 103 to a capturedimage which is described later, and visualizes mutual positionmisalignment.

From the position misalignment image, a display information emphasizingunit 111 extracts an image of only the fixed part 104 and the targetpart 103 that should be subject to emphasized display, magnifies, andgenerates an emphasized image that is emphasized display information. Adisplay composition unit 112 generates a composite image (compositedisplay information) by composing the emphasized image and, asnon-emphasized display information, a region that excludes an imagecorresponding to the target part 103 and the fixed part 104 in theposition misalignment image, achieving matching so that a sense ofunnaturalness does not occur between the two. A display output unit 113is connected to a display unit 114 such as a liquid crystal monitor, andoutputs the generated composite image to a viewer so that it can beseen.

Note that, other than components shown in the figure, the systemillustrated in FIG. 1 is provided with components installed in acomputer, such as a CPU, a ROM, and a RAM. By the CPU reading a programstored in the ROM and executing it on the RAM, operations of variouscomponents illustrated in FIG. 1 are controlled. In addition, variouscomponents of the image input unit 108 to the display output unit 113may be realized by dedicated hardware, or may be realized by a program(software) executed by the CPU.

FIG. 2 illustrates a flowchart for describing an operation by which acaptured image from the camera 105 mounted on the leading end of therobot 101 is used for visualization, after emphasizing positioninformation necessary for teaching. Note that processing that thisflowchart illustrates is realized by the CPU reading and executing aprogram stored in the ROM.

In step S201, the camera 105 captures a state/condition of the finger102 positioned in front of the camera 105, the target part 103 that thefinger 102 grips, and the fixed part 104. An example of a captured imagethereby obtained is a captured image 301 illustrated in FIG. 3. Asillustrated in FIG. 3, a state in which images of each of the finger102, the target part 103, and the fixed part 104 are included in thecaptured image 301 is illustrated.

In step S202 the image input unit 108 obtains the captured image as aninput image, and holds it. In step S203 the measurement informationobtaining unit 109 obtains measurement information. Specifically, asillustrated in FIG. 4, the measurement information obtaining unit 109,with the tracker 106 as a basis position, obtains as measurementinformation a position/orientation Pc of the target coil 107 that isattached to a leading end of the robot 101. As described below, theposition/orientation Pc is measured by using the tracker 106.

The tracker 106 is provided internally with transmission coils in eachof three orthogonal axis directions, and causes an alternating currentmagnetic field of a predetermined frequency to be generated in a spacethereof. The target coil 107 is also provided internally with receptioncoils in each of three orthogonal axis directions, and generateselectromotive force due to an electromagnetic induction in thealternating current magnetic field. By comparing a magnitude of theelectromotive force of the reception coils overall with values at eachposition that are measured in advance, it is possible to measure adistance between the tracker 106 and the target coil 107. In addition,by comparing a magnitude of the electromotive force of each axis of thereception coils when the magnetic field is generated at each axis of thetransmission coils, it is possible to measure an orientation at thethree axes of the reception coils, in other words, the target coil 107.

The tracker 106 uses these measurement values to measure aposition/orientation Pc of the target coil 107. As in Equation (1)below, Pc is represented by a 4×4 (four-by-four) affine matrix thatrespectively combines a rotation matrix Rc and a translation vector Tc.Below, positions and orientations described similarly are represented bysimilar affine matrices. After measurement by the tracker 106, theposition/orientation Pc is transferred to the measurement informationobtaining unit 109 as measurement information, and held.

$\begin{matrix}\left\lbrack {{EQUATION}\mspace{14mu} 1} \right\rbrack & \; \\{P_{c} = {\left( {R_{c}T_{c}} \right) = \begin{pmatrix}R_{cxx} & R_{cyx} & R_{czx} & T_{cx} \\R_{cxy} & R_{cyy} & R_{czy} & T_{cy} \\R_{cxz} & R_{cyz} & R_{czz} & T_{cz} \\1 & 1 & 1 & 1\end{pmatrix}}} & (1)\end{matrix}$

The measurement information obtaining unit 109 holds, as environmentinformation, the position/orientation Pa of the camera 105 and theposition/orientation Pf of the leading end of the finger 102, which havethe target coil 107, measured in advance, as the basis position.Furthermore, the measurement information obtaining unit 109 holds, asenvironment information, an insert position/orientation Pt of the targetpart 103 with the position/orientation Pf as a basis, and an insertposition/orientation Px of the fixed part 104 with the tracker 106 as abasis position. In addition, the measurement information obtaining unit109 holds a focal distance f, which is a camera intrinsic parameter ofthe camera 105 and is measured in advance, and CAD data, which isthree-dimensional shape information of the target part 103 and the fixedpart 104. Note that an origin position of the camera 105 used in thepresent embodiment and a center of an image are assumed to match.

In step S204 the display information conversion unit 110 uses thecaptured image 301 as well as the environment information and themeasurement information that the measurement information obtaining unit109 holds to generate the display information. FIG. 5 illustrates aposition misalignment image 501, which is an example of the displayinformation.

Initially, the display information conversion unit 110 obtains theposition/orientation Pc, which is measurement information, from themeasurement information obtaining unit 109. Similarly, the displayinformation conversion unit 110 obtains each kind of thepositions/orientations Pa, Pf, Pt, and Px, which are environmentinformation, from the measurement information obtaining unit 109. Inaddition, the display information conversion unit 110 also obtains, fromthe measurement information obtaining unit 109, the focal distance f ofthe camera 105, which is measured in advance, as well as the CAD data ofthe target part 103 and the fixed part 104.

Next, the display information conversion unit 110 calculates an insertposition Ttc of the target part 103 and an insert position Txc of thefixed part 104, on the captured image viewed from the camera 105. Ttcand Txc can be calculated by using the position and orientation of eachcomponent in the following Equation (2).

$\begin{matrix}\left\lbrack {{EQUATION}\mspace{14mu} 2} \right\rbrack & \; \\\left\{ \begin{matrix}{T_{tc} = {{Proj} \cdot P_{a}^{- 1} \cdot P_{f} \cdot P_{t} \cdot \begin{pmatrix}0 \\0 \\0\end{pmatrix}}} \\{T_{xc} = {{Proj} \cdot P_{a}^{- 1} \cdot P_{c} \cdot P_{x} \cdot \begin{pmatrix}0 \\0 \\0\end{pmatrix}}}\end{matrix} \right. & (2)\end{matrix}$

Here, Proj in Equation (2) is a projection matrix of the camera 105, andis represented in the following Equation (3) by using the focal distancef.

$\begin{matrix}\left\lbrack {{EQUATION}\mspace{14mu} 3} \right\rbrack & \; \\{{Proj} = \begin{pmatrix}f & 0 & 0 & 0 \\0 & f & 0 & 0 \\0 & 0 & 1 & 0\end{pmatrix}} & (3)\end{matrix}$

The display information conversion unit 110 renders, on the capturedimage 301, crosshair cursors 502 and 503 on positions of the calculatedinsert positions Ttc and Txc, as illustrated in FIG. 5. These functionas display information that visualize misalignment of the insertpositions of the target part 103 and the fixed part 104. Furthermore,the display information conversion unit 110 renders an edge portion 504of a three-dimensional shape of each of the target part 103 and thefixed part 104, as display information, on top of the insert positionsTtc and Txc. At this point, a hidden edge portion that is hidden fromthe viewpoint of the camera 105 and cannot be observed directly isrendered with a dotted line. With this, even if an appearance of thefixed part 104 is largely obstructed due to the target part 103, it ispossible to visually recognize the position of the fixed part 104.

In step S205, the display information emphasizing unit 111 generates anemphasized image for emphasizing display information. Specifically, thedisplay information emphasizing unit 111 extracts from the positionmisalignment image 501 and magnifies only an inserting unit 505 of thefixed part 104 and the target part 103 that should be subject toemphasized display, and generates, as emphasized display information, anemphasized image 601 as illustrated in FIG. 6.

Firstly, from a region for which a three-dimensional shape of the targetpart 103 and the fixed part 104 was rendered on the insert positions Ttcand Txc calculated in step S203, an emphasized target region 701 (FIG.7) corresponding to the target part 103 and the inserting unit 505 ofthe fixed part 104 in the position misalignment image 501 of FIG. 5 isextracted. Next, centering the extracted emphasized target region 701 ona center-point position Tcenter of the insert positions Ttc and Txc, forexample, expansion with a magnification factor M=1.5 times is performed,and the emphasized image 601 illustrated in FIG. 6 is generated.

With this, by magnifying and emphasizing minute position misalignmentbetween the target part 103 and the fixed part 104 for whichconfirmation was difficult on the position misalignment image 501, it ispossible to easily confirm understanding of the position misalignment.

In step S206, the display composition unit 112 performs displaycomposition of the position misalignment image 501 and the emphasizedimage 601. Specifically, the display composition unit 112 composes theposition misalignment image 501, which is non-emphasized displayinformation, and the emphasized image 601, which is emphasized displayinformation, by causing them to be matching such that a sense ofunnaturalness does not arise. FIG. 8 illustrates an example of acomposite image 801.

Here, firstly the display composition unit 112 extracts a matchingregion 901, illustrated in FIG. 9, as a difference between a region thatmagnifies by 1.7 times the emphasized target region 701 in the positionmisalignment image 501 converted in step S204, and a region of theemphasized target region 701. Next, the display composition unit 112generates a matching image 1001 that is illustrated in FIG. 10 byperforming deformation of the image in the matching region 901, whereinthe magnification factor M of a region closest to the center Tcenter isequally set as 1.5 times, a farthest region is set to no magnification,and the magnification factor therebetween is changed in a gradation.Furthermore, the display composition unit 112 generates the compositeimage 801 by superimposing the matching image 1001 and the emphasizedimage 601 in that order on the position misalignment image 501. Withthis, it is possible to perform unification with no sense ofunnaturalness and without a discontinuous portion occurring between theposition misalignment image 501 and the emphasized image 601.

While having a field of view that does not change from the originalcaptured image, the composite image 801 enables a minute positionmisalignment between the target part 103 and the fixed part 104, towhich attention should be paid, to be easily understood. As a result, aviewer intuitively can understand information of position misalignmentfrom the composite image, without the screen content being complicated.In other words, in the present embodiment, by generating the emphasizedimage 601, it is possible to make a predetermined region of interest(the emphasized image 601) in the image easily distinguishable fromanother region. Note, in this embodiment, by magnifying a region ofinterest image in a display image or making a density value (a luminancevalue/a pixel value) thereof higher than another region image, anemphasized image that emphasizes a characteristic of an image isgenerated, but limitation is not made to this. For example, instead of adensity value, configuration may be taken to apply a predeterminedpattern or to apply sharpening to a target region of interest image.

In step S207 the display output unit 113 outputs the image.Specifically, the composite image 801 illustrated in FIG. 8 is output onthe display unit 114, and is presented to a viewer. Through the outputof the display unit 114, the viewer can confirm position misalignment ofthe robot at a time of teaching.

As explained above, by virtue of the present embodiment, a viewer canconfirm intuitively and easily position misalignment by observing acomposite image that emphasizes a displacement and a size of a robot ora portion of a target that the robot operates, in an image captured by acamera.

Second Embodiment

In the second embodiment, instead of obtaining information regarding anactual robot according to the first embodiment, explanation is given fora robot information display system, in an offline teaching environmentreproduced by a computer, that emphasizes and then visualizes positioninformation necessary at a time of teaching by using CG (ComputerGraphics) images.

In a conventional offline teaching environment, information such asangle misalignment or position misalignment in each axial direction isalso superimposed, with a graph or a size of an arrow or a numericalvalue, onto a CG image that represents a robot environment. However, ifmany pieces of information are superimposed, there is a problem in thatscreen content becomes complicated, and it is hard to see minutemisalignments.

Accordingly, in the present embodiment, visualizing is performed afteremphasizing a displacement and a size of a robot or a portion of atarget that the robot operates in a CG image, and output to a viewer isperformed. With this, by observing a composite image that uses the CGimage, the viewer can confirm intuitively and easily positionmisalignment at a time of an offline teaching for a robot.

FIG. 11 illustrates an overall configuration of a robot informationdisplay system that emphasizes minute position misalignment in a CGimage of a part that a robot grips and inserts; and that outputs to aviewer.

An environment reproduction unit 1101 reproduces by simulation athree-dimensional shape and mechanical behavior of the robot 101according to the first embodiment and a surrounding environment, andvirtually establishes a reproduction environment 1301 illustrated inFIG. 13. A CG generator 1102 generates, based on the virtuallyestablished reproduction environment 1301, a CG image of a viewpointviewed from a virtual camera 1304 illustrated in FIG. 13.

An image input unit 1108 through to a display unit 1114 in FIG. 11respectively correspond to the image input unit 108 to the display unit114 in FIG. 1 of the first embodiment, and have the same functions.However, the image input unit 1108 obtains a CG image as the inputimage. In addition, in place of obtaining the measurement informationfrom the tracker 106 in the first embodiment, a measurement informationobtaining unit 1109 obtains the measurement information from theenvironment reproduction unit 1101.

Note that, other than components shown in the figure, the systemillustrated in FIG. 11 is provided with components installed in acomputer, such as a CPU, a ROM, and a RAM. By the CPU reading a programstored in the ROM and executing it on the RAM, operations of variouscomponents illustrated in FIG. 11 are controlled. In addition, variouscomponents of the environment reproduction unit 1101 to a display outputunit 1113 may be realized by dedicated hardware, or may be realized by aprogram (software) executed by the CPU.

FIG. 12 illustrates a flowchart for describing operations that outputhaving emphasized the position information necessary for teaching withrespect to a CG image displayed at a time of offline teaching. Note thatprocessing that this flowchart illustrates is realized by the CPUreading and executing a program stored in the ROM.

In step S1201 the environment reproduction unit 1101 generates areproduction environment. Specifically, the environment reproductionunit 1101 reproduces by simulation a three-dimensional shape andmechanical behavior of the robot 101 according to the first embodimentand a surrounding environment, and virtually establishes thereproduction environment 1301 illustrated in FIG. 13. The reproductionenvironment 1301 is comprised from CAD data that represents shapes of avirtual robot 1302, a virtual finger 1303, the virtual camera 1304, avirtual target part 1305, and a virtual fixed part 1306 which areillustrated in FIG. 13; positions and orientations respectivelycorresponding to Pa, Pf, Pt and Px in FIG. 4 of the first embodiment;and respective joint angles of the virtual robot 1302 for generating aposition and orientation corresponding to the position/orientation Pc ofthe target coil 107.

In step S1202 the CG generator 1102 generates, based on information ofthe generated reproduction environment 1301, a CG image of the virtualrobot 1302 and the surrounding environment thereof, from the viewpointof the virtual camera 1304 installed on the virtual robot 1302.

Next, ensuing step S1203 to step S1206 perform operations similar tothose in step S203 to step S206 in the first embodiment. However, instep S202 of the first embodiment, the captured image 301 is the inputimage, but in step S1202 instead of the captured image 301 the CG imageis the input image. In addition, in step S203 of the first embodiment,measurement information is obtained from the tracker 106, but in stepS1203, instead of the tracker 106, measurement information is obtainedfrom the environment reproduction unit 1101 that establishes thereproduction environment 1301.

As explained above, by virtue of the present embodiment, a viewer canconfirm intuitively and easily position misalignment by observing acomposite image that emphasizes a displacement and a size of a robot ora portion of a target that the robot operates, in a CG image that isgenerated.

Third Embodiment

In the third embodiment, similarly to the first embodiment, explanationis given for a robot information display system in which it is possibleto, after viewing an output composite image, change an operation gain ofa robot in accordance with a degree of emphasis of emphasized positionmisalignment, to actually operate the robot.

In a conventional system, there is a form in which, with respect to anoperation by a teaching pendant, a robot operates by a fixed operationgain set by an teacher. However, when it is far from a target position,the operation gain is insufficient, and when it is close to the targetposition there is a necessity to decrease the operation gain, andmoreover, to accurately align to a target value it is necessary tofinely observe movement of the robot and confirm position misalignment.

In the present embodiment, by emphasizing a displacement or size of aportion of the robot or a target that the robot operates in the capturedimage, a viewer can easily confirm position misalignment. Furthermore,based on a difference between a target value position and a currentposition, the degree of emphasis of the position misalignment is causedto change automatically, and the degree of emphasis and the operationgain of the robot are caused to interwork. Thereby, it is possible tocause the robot to operate at high speed if far from the targetposition, and at low speed as the target position is approached, and itis possible to teach a more complicated and delicate operation simplyand in a short time.

FIG. 14 illustrates a configuration of a robot information displaysystem as a whole, in which an action force generated when parts that arobot grips and performs inserting for contact each other is emphasizedin a captured image and output to a viewer, and operation gain of therobot is changed in accordance with a degree of emphasis.

A robot 1401 through to a display unit 1414 in FIG. 14 respectivelycorrespond to the robot 101 to the display unit 114 in FIG. 1 of thefirst embodiment, and have the same functions. However, the robot 1401operates in accordance with an operation signal from a robot operationunit (for example, an operation board) 1417. In addition, a displayinformation emphasizing unit 1411 generates an emphasized image based ona degree of emphasis set by a degree-of-emphasis setting unit 1415.

The degree-of-emphasis setting unit 1415 sets a magnification factor,which is the degree of emphasis of the emphasized image generated by thedisplay information emphasizing unit 1411, based on measurementinformation of the position/orientation Pc of a target coil 1407, andenvironment information of the robot 1401 and a periphery thereof, suchas a position of a fixed part 1404 or a positional relationship, held bya position/orientation obtaining unit 1409, between the target coil 1407and a finger 1402 of the robot 1401.

An operation gain setting unit 1416 sets the operation gain of the robotbased on the magnification factor set by the degree-of-emphasis settingunit 1415. The robot operation unit 1417 functions as a robot controllerthat controls the operation of the robot by transmitting a controlparameter to the robot 1401, based on the operation gain set by theoperation gain setting unit 1416 and an operation of a viewer. The robotoperation unit 1417 is comprised by a physical operation component, suchas an operation button, an operation controller, or a switch.

Note that, other than components shown in the figure, the systemillustrated in FIG. 14 is provided with components installed in acomputer, such as a CPU, a ROM, and a RAM. By the CPU reading a programstored in the ROM and executing it on the RAM, operations of variouscomponents illustrated in FIG. 14 are controlled. Each kind ofcomponents of an image input unit 1408 through to a display output unit1413 and the degree-of-emphasis setting unit 1415 and the operation gainsetting unit 1416 may be realized by dedicated hardware, or may berealized by a program (software) executed by a CPU.

FIG. 15 illustrates a flowchart for describing operations in which acaptured image from a camera mounted on a robot leading end is used,visualizing is performed by emphasizing position information necessaryfor teaching, an operation gain of the robot is set in accordance with adegree of emphasis thereof, to operate the robot. Note that processingthat this flowchart illustrates is realized by the CPU reading andexecuting a program stored in the ROM.

Step S1501 to step S1504 perform operations similar to those in stepS201 to step S204 in the first embodiment. In step S1505, thedegree-of-emphasis setting unit 1415 sets a degree of emphasis thatindicates a degree of emphasis with respect to the position misalignmentimage. In particular, the degree-of-emphasis setting unit 1415 sets amagnification factor M as the degree of emphasis. The magnificationfactor M is represented by the following Equation (4).

$\begin{matrix}\left\lbrack {{EQUATION}\mspace{14mu} 4} \right\rbrack & \; \\{M = \frac{\alpha}{\sqrt{T_{x} - T_{tfc}} + \beta}} & (4)\end{matrix}$

Here, α is a coefficient holding a fixed value for adjusting the degreeof the magnification factor M, and β is a coefficient having a fixedvalue for setting a maximum value of the magnification factor to M=α/βwhen the position misalignment is infinitesimal. Tx and Ttfc in Equation(4) are represented by the following Equations (5).

$\begin{matrix}\left\lbrack {{EQUATION}\mspace{14mu} 5} \right\rbrack & \; \\\left\{ \begin{matrix}{P_{x} = \left( {R_{x}T_{x}} \right)} \\{P_{tfc} = {\left( {R_{tfc}T_{tfc}} \right) = {P_{t} \cdot P_{f} \cdot P_{c}}}}\end{matrix} \right. & (5)\end{matrix}$

Pc, Pf, Pt, and Px are the same as respective positions and orientationsin the first embodiment, and are obtained from the position/orientationobtaining unit 1409. In this way, by causing the magnification factor Mto change dynamically and automatically, it is possible to easilyconfirm minute position misalignment without performing emphasizingparticularly if far from the target position, and increasing the degreeof emphasis as the target position is approached.

Step S1506 to step S1508 perform operations similar to those in stepS205 to step S207 in the first embodiment. In step S1509 the operationgain setting unit 1416 sets an operation gain G of the robot 1401 basedon the set magnification factor M. The operation gain G of the robot1401 is set as G=1 at a standard time, and is set with a ratio of a unitmovement amount at a time of an operation input of the robot 1401. Usingthe magnification factor M, the operation gain G is caused todynamically change in accordance with the magnification factor M, asexpressed by the following Equation (6).

$\begin{matrix}\left\lbrack {{EQUATION}\mspace{14mu} 6} \right\rbrack & \; \\{G = \frac{\gamma}{M}} & (6)\end{matrix}$

Here γ is a proportionality coefficient that has a fixed value forappropriately adjusting a value of the operation gain G with respect tothe magnification factor M, and is expressed by Equation (7) so that G=1when the magnification factor M is a maximum value.

$\begin{matrix}\left\lbrack {{EQUATION}\mspace{14mu} 7} \right\rbrack & \; \\{\gamma = \frac{\alpha}{\beta}} & (7)\end{matrix}$

In step S1509 the robot operation unit 1417 receives a robot operationinput in accordance with a button input by a viewer, and causes therobot 1401 to operate. Regarding the operation of the robot 1401, with atracker 1406 as a basis position, a rotational angle and athree-dimensional position in XYZ directions of an end effector of therobot 1401 are caused to be changed by a viewer's button input. A unitmovement amount S of the robot in accordance with one button operationinput is expressed by the following Equation (8).[EQUATION 8]S=Gs  (8)

Here, a base movement amount s is a basis maximum speed, and apredetermined speed is set in advance for when causing it to operate athigh speed if the operation gain G=1, in other words if the robot issufficiently far from the target position. With such a configuration, bycausing the unit movement amount S to change in accordance with theoperation gain G, it is possible to cause the robot to operate a highspeed when far from the target position and at low speed as itapproaches the target position, and it is possible to perform alignmentto the target value by operating the robot swiftly and accurately.

As explained above, by virtue of the present embodiment, by emphasizinga displacement or size of a portion of the robot or a target that therobot operates in the captured image, a viewer can easily confirm aposition misalignment. Furthermore, based on a difference between atarget value position and a current position, the degree of emphasis iscaused to change automatically, and the degree of emphasis and theoperation gain of the robot are caused to interwork. Thereby, it ispossible to cause the robot to operate at high speed if far from thetarget position, and at low speed as the target position is approached,and it is possible to teach a more complicated and delicate operationsimply and in a short time.

Other Embodiments

<Setting a Plurality of Locations and Priorities and Emphasizing>

For composite images according to the first to third embodiments,emphasizing is performed for position information of one place, butthere is no limitation to this. For example, in a case such as when atarget object is caused to pass through a vicinity of an obstaclewithout causing it to contact the obstacle, configuration may be takento magnify and emphasize a plurality of positions of parts for whichcontact is possible.

Note that, there is a problem in that, if too many locations areemphasized, the composite image will be greatly changed from thecaptured image, it will become difficult to confirm a region that is notemphasized, and intuitive understanding of the position information willbecome difficult. For this reason, if many positions that are candidatesfor emphasizing are present, configuration may be taken so as to set apriority order or restrict an emphasizing target by adding, for example,to the configuration of the third embodiment an emphasized partselecting unit, which has a function of selecting a position (part) toemphasize. At this point, a teaching order, by which the robot performsalignment with respect to task content that is its objective, is set inadvance for the emphasized part selecting unit, and by a viewerdesignating a current teaching order in accordance with the robotoperation unit, it is possible to cause automatic selection of aposition to emphasize. Of course, configuration may be taken to providein the robot operation unit an emphasized position designating unit,such as a trackpad, that designates a position in the captured image,and directly instruct a position to emphasize.

With this, by limiting portions to emphasize in the composite image,intuitive understanding of the position information is possible, even ifthere are many candidate positions for emphasizing.

<When Deforming in Accordance with Action Force>

For emphasized images in the first to third embodiments, emphasizing isperformed for position information necessary when teaching, but there isno limitation to this. In addition, with respect to a displacementamount or an amount of change, such as that of an angle, a speed, anacceleration, or an action force, if it is information for whichemphasizing in accordance with magnification, movement, or deformationis possible, any such information may be used. For example, a forcesensor is added to the robot and an action force generated by an actionsuch as a contact between the target part and the fixed part ismeasured, and this may be visualized and emphasized. FIGS. 16A and 16Billustrate a pre-emphasis image (FIG. 16A) and a post-emphasis image(FIG. 16B) for an action force f generated by contact on a contact point1605 when inserting a target part 1602 gripped by a finger 1601 of therobot into an insert position 1604 on a fixed part 1603.

In such a case, assuming that the target part 1602 is weak in terms ofstrength and is an easy to break object, the target part 1602, to whichmore attention should be paid, is emphasized by reducing in a directionthat the action force f occurs. Of course, configuration may be taken todeform both in proportion to a strength ratio or a hardness, or if ashear force due to friction is occurring, and configuration may be takento deform diagonally in a shear direction instead of reducing.

With this, it is possible to understand intuitively the size andposition of an occurrence of an action force which originally could notbe confirmed by only viewing the captured image.

<When Emphasizing Angle Misalignment>

As another example, configuration may be taken such that if misalignmentoccurs in a rotational angle between the target part and the fixed part,this is emphasized. FIGS. 17A and 17B illustrate a pre-emphasizing image(FIG. 17A) and a post-emphasizing image (FIG. 17B) of an anglemisalignment θ that occurs when teaching an inserting operation of arectangular target part 1702 gripped by a finger 1701 of the robot intoa fixed part 1703.

The target part 1702 is inclined θ=2 degrees with respect to aninserting angle of the fixed part 1703, and as is inserting cannot beperformed; however, because angle misalignment is very small, asillustrated in FIG. 17A, confirming misalignment by visual observationis difficult as is. Accordingly, as illustrated in FIG. 17B, by settingthe angle misalignment to θ′=20 degrees which is 10 times bigger, theangle misalignment is caused to be emphasized. At this point, theinclination of the finger 1701, which grips the target part 1702, issimilarly changed. By changing the angle of the finger 1701, the displaycomposition unit achieves matching of angles of other joints of therobot as necessary, so that a sense of unnaturalness does not occur in acomposite image, and then outputs as the composite image.

With this, it is possible to understand intuitively minute anglemisalignment between components.

<When Deforming by Acceleration or Speed>

As another example, if a movement speed or an acceleration of the robotis important as information, configuration may be taken to visualize andemphasize this by emphasizing a size of the target part, a finger of therobot, or the like. FIGS. 18A and 18B illustrate a pre-emphasis image(FIG. 18A) and a post-emphasis image (FIG. 18B) for the movement speed,at a time of an operation to move in a movement direction 1803 by avelocity v while a target part 1802 is held by a finger 1801 of therobot.

In the present example, the finger 1801 and the target part 1802, whichare targets to be emphasized, are emphasized by magnifying in proportionto the velocity v in the movement direction 1803. Of course, ifacceleration instead of the velocity v is important as information,emphasis may be performed in accordance with the acceleration. Inaddition, in addition to magnifying, a video effect such as outputtingan afterimage along a path of movement may be performed.

With this, even in a situation such as where a pattern in a backgroundis deficient and a relative movement speed is difficult to understand,it is possible to understand intuitively a magnitude or a direction of aspeed at which the robot moves.

<Automatic/Manual Adjustment of a Degree of Emphasis>

The magnification factor M of the composite image in the thirdembodiment is automatically set by dynamically using Equation (4), butno limitation is made to this. Any configuration may be used if it is aconfiguration that automatically or manually sets the magnificationfactor M. For example, configuration may be taken such that a knob or alike, which is an emphasis degree adjusting unit, is added to the robotoperation unit, and a viewer can operate a magnification factordirectly. With this, although operations increase, it is possible toperform emphasis as much as a viewer intends. Of course, similarly tothe third embodiment, the operation gain G of the robot may be changedin accordance with the magnification factor M set manually.

<Method of Emphasizing and Achieving Matching when Rendering CG>

For composite images in the second embodiment, emphasizing is performedusing a CG image generated in step S1202, but there is no limitation tothis. If it is possible to finally output as CG a composite image basedon CAD data, a configuration that implements other emphasizing may beused. For example, instead of generating the CG image in step S1203,three-dimensional shapes in the CAD data of the virtual target part 1305and the virtual fixed part 1306 obtained from the measurementinformation obtaining unit 1109 are held in advance, and in step S1204only portions corresponding to emphasized target regions of the virtualtarget part 1305 and the virtual fixed part 1306 are extracted,magnified in a three-dimensional shape stage, and caused to beemphasized.

Configuration may be taken to use a form in which the display outputunit 1113 causes a CG image to be generated in step S1206 whileobtaining a match by causing the magnification factor of athree-dimensional shape of an emphasized target region periphery (withina predetermined area centered on a position (e.g., a centroid) thatdefines the emphasized target region) to change in step S1205.

In this way, by causing magnification/emphasizing as a three-dimensionalshape, it is possible to generate a composite image that is lessinconsistent as an image, and for which it is easy to intuitivelyunderstand position misalignment.

<Method of Obtaining a Match by Using Degrees of Freedom Originally notMovable>

In step S1205, by processing in which the display composition unit 1105obtains a match, the magnification factor of a three-dimensional shapeof an emphasized target region periphery is caused to be changed, butlimitation is not made to this. If there is a configuration by which itis possible to compose the emphasized display information and thenon-emphasized display information without a sense of unnaturalness,other processing that achieves matching may be performed. For example,instead of causing the magnification factor to be changed, thethree-dimensional shape of the emphasized target region periphery may becaused to be translated or rotated. For example, when the virtual targetpart 1305 is magnified, as is the virtual finger 1303 of the emphasizedtarget region periphery that grips the virtual target part 1305, whichis the emphasized target region, enters a state in which misalignmentwith a gripping position occurs as illustrated in FIG. 19A. Here,instead of a translational direction 1901 in which the virtual finger1303 originally moved, as illustrated in FIG. 19B, rotation is causedfor a rotation direction 1902 of a base, and it is made to be matchingwith the gripping position.

With such a configuration, by causing the emphasized target regionperiphery of the robot to move to a position or in a rotation directionin which there is no operation originally, it is possible to generate acomposite image that has no inconsistency as an image and by which it iseasy to intuitively understand position misalignment.

<Method of Explicitly not Dividing Emphasized Target Region andNon-Emphasized Target Region>

By processing that achieves matching in step S206 of the firstembodiment for example, the display composition unit in the first tothird embodiments sets a matching region between the emphasized displayinformation and the non-emphasized display information to achievematching, but limitation is not made to this. If there is aconfiguration by which it is possible to compose the emphasized displayinformation and the non-emphasized display information without a senseof unnaturalness, other processing that achieves matching may beperformed. For example, configuration may be taken to set a maximummagnification factor M in a central portion of the emphasized displayinformation, and achieve matching by gradually causing the magnificationfactor M to decrease as a region corresponding to the non-emphasizeddisplay information is approached.

With such a configuration, by causing a degree of emphasis to changecontinuously with respect to the non-emphasized display informationhaving centered on the emphasized display information, it is possible togenerate a composite image for which there is less inconsistency as animage.

Effect of Embodiments

In the first embodiment, the captured image from the camera mounted on arobot leading end is used, visualization after emphasizing positioninformation necessary at time of teaching is performed, and outputtingto a viewer is performed. With this, by observing a composite image thatuses the image captured by the camera, the viewer can confirm positionmisalignment intuitively and easily.

In the second embodiment, in an offline teaching environment reproducedby a computer in place of an actual robot, a CG image is used to performvisualizing that emphasizes the position information necessary at a timeof teaching, and outputting to a viewer is performed. With this, byobserving a composite image that uses the CG image, the viewer canconfirm position misalignment intuitively and easily.

In the third embodiment, similarly to the first embodiment, afterviewing an output composite image, an operation gain of a robot ischanged in accordance with a degree of emphasis of emphasized positionmisalignment, to actually operate the robot. With this, at a time ofteaching the robot, a viewer can easily confirm position misalignment.Furthermore, it is possible to cause the robot to operate at high speedif far from the target position, and at low speed as the target positionis approached, and it is possible to teach a more complicated anddelicate operation simply and in a short time.

Definitions

The measurement information obtaining unit may be anything if it holdsmeasurement information by any of one or more measurement scales(measurement references) of a position, an orientation, a speed, anacceleration, or an action force of a processing target that includes atleast one of the robot and a target that the robot operates. An exampleis the measurement information obtaining unit according to the first tothird embodiments.

In addition, the display information conversion unit may be anything ifit can convert the measurement information into information that isviewable by a viewer. An example is the display information conversionunit in the first and third embodiments, which renders, with respect tothe captured image, edges of a component and a crosshair cursor onpositions of the target part and the fixed part. Also, another exampleis the display information conversion unit in the second embodiment,which renders, with respect to a CG image, edges of a component and acrosshair cursor on positions of the target part and the fixed part.

In addition, the display information emphasizing unit may be anything ifit emphasizes a feature (displacement or size) of a part of theprocessing target, which includes at least one of the robot and a targetthat the robot operates. An example is the display informationemphasizing unit in the first to third embodiments that magnifies andemphasizes the position misalignment between the target part and thefixed part.

The display composition unit may be anything if it composes by achievingmatching between the emphasized display information, which is emphasizedby the display information emphasizing unit, and the non-emphasizeddisplay information, which is other than the emphasized displayinformation in the display information and is not emphasized. An exampleis the display composition unit in the first to third embodiments thatunifies by achieving matching by providing a matching region between theemphasized display information and the non-emphasized displayinformation so that a sense of unnaturalness is not present.

In addition, the display output unit may be anything if it outputscomposite display information composed by the display composition unitto a viewer. An example is the display output unit in the first to thirdembodiments.

In addition, an information holding unit, an information conversionunit, a display information emphasizing unit, and a display compositionunit, may be a form in which a storage medium storing program code ofsoftware that realizes functions thereof is supplied to a system or anapparatus.

Embodiment(s) of the present invention can also be realized by acomputer of a system or apparatus that reads out and executes computerexecutable instructions (e.g., one or more programs) recorded on astorage medium (which may also be referred to more fully as a‘non-transitory computer-readable storage medium’) to perform thefunctions of one or more of the above-described embodiment(s) and/orthat includes one or more circuits (e.g., application specificintegrated circuit (ASIC)) for performing the functions of one or moreof the above-described embodiment(s), and by a method performed by thecomputer of the system or apparatus by, for example, reading out andexecuting the computer executable instructions from the storage mediumto perform the functions of one or more of the above-describedembodiment(s) and/or controlling the one or more circuits to perform thefunctions of one or more of the above-described embodiment(s). Thecomputer may comprise one or more processors (e.g., central processingunit (CPU), micro processing unit (MPU)) and may include a network ofseparate computers or separate processors to read out and execute thecomputer executable instructions. The computer executable instructionsmay be provided to the computer, for example, from a network or thestorage medium. The storage medium may include, for example, one or moreof a hard disk, a random-access memory (RAM), a read only memory (ROM),a storage of distributed computing systems, an optical disk (such as acompact disc (CD), digital versatile disc (DVD), or Blu-ray Disc (BD)™),a flash memory device, a memory card, and the like.

While the present invention has been described with reference toexemplary embodiments, it is to be understood that the invention is notlimited to the disclosed exemplary embodiments. The scope of thefollowing claims is to be accorded the broadest interpretation so as toencompass all such modifications and equivalent structures andfunctions.

This application claims the benefit of Japanese Patent Application No.2015-005287, filed Jan. 14, 2015, which is hereby incorporated byreference herein in its entirety.

What is claimed is:
 1. A display control apparatus configured to displayan operation of a robot, the display control apparatus comprising: atleast one processor; and at least one memory coupled with the at leastone processor and storing instructions that, when executed by the atleast one processor, cause the display control apparatus to: obtainmeasurement information of a processing target that includes at leastone of the robot and a target that the robot operates; generate, basedon the measurement information, display information that is viewable bya viewer; generate emphasized display information that emphasizes afeature of a portion of the processing target by magnifying or changingangle of or deforming the processing target, based on the displayinformation; generate composite display information in which theemphasized display information and non-emphasized display informationother than the emphasized display information are composed, thecomposite display information being generated by matching the emphasizeddisplay information and the non-emphasized display information in thedisplay information; and output the composite display information to adisplay device.
 2. The display control apparatus according to claim 1,wherein the instructions that, when executed by the at least oneprocessor, further cause the display control apparatus to: input animage of the processing target, wherein the display information, whichis based on the measurement information, is generated by processing theinput image of the processing target based on the measurementinformation.
 3. The display control apparatus according to claim 2,wherein the instructions that, when executed by the at least oneprocessor, further cause the display control apparatus to: image theprocessing target, wherein the imaged processing target is input as theimage of the processing target.
 4. The display control apparatusaccording to claim 2, wherein the instructions that, when executed bythe at least one processor, further cause the display control apparatusto: generate a computer graphics (CG) image of the processing target,wherein the generated CG image of the processing target is input as theimage of the processing target.
 5. The display control apparatusaccording to claim 1, wherein at a position at which the measurementinformation of the processing target changes, the feature of the portionof the processing target is emphasized by causing a magnitude of theprocessing target in a predetermined area centered on the position tochange.
 6. The display control apparatus according to claim 1, whereinthe feature of the portion of the processing target is emphasized bymaking a displacement amount of the measurement information of theprocessing target larger than in reality.
 7. The display controlapparatus according to claim 1, wherein the feature of the portion ofthe processing target is emphasized by, in accordance with a speed or anacceleration of the processing target, magnifying a magnitude of theprocessing target in a direction that the speed or the acceleration isoccurring.
 8. The display control apparatus according to claim 1,wherein the feature of the portion of the processing target isemphasized by causing a magnitude of a shape of the processing target todeform in accordance with an action force.
 9. The display controlapparatus according to claim 1, wherein the instructions that, whenexecuted by the at least one processor, further cause the displaycontrol apparatus to: select a portion that is emphasized, if aplurality of candidates of the portion of the processing target that isemphasized are present.
 10. The display control apparatus according toclaim 9, wherein the portion to be emphasized is selected based onpredetermined task content that is an objective of the robot, anoperation by a viewer, or both.
 11. The display control apparatusaccording to claim 1, wherein the instructions that, when executed bythe at least one processor, further cause the display control apparatusto: dynamically change a degree of emphasis.
 12. The display controlapparatus according to claim 11, wherein the degree of emphasis isdynamically changed based on a difference between a current value and apredetermined target value for the feature of the processing target. 13.The display control apparatus according to claim 11, wherein the degreeof emphasis is dynamically changed in accordance with an operation froma viewer.
 14. The display control apparatus according to claim 1,wherein the instructions that, when executed by the at least oneprocessor, further cause the display control apparatus to: control theoperation of the robot by transmitting a control parameter to the robot,wherein the control parameter causes an operation gain of the robot tochange in accordance with a degree of emphasis of the feature, which isemphasized.
 15. The display control apparatus according to claim 1,wherein the emphasized display information and the non-emphasizeddisplay information are composed by causing a degree of emphasis betweenthe emphasized display information and the non-emphasized displayinformation to change continuously.
 16. A display control method fordisplaying an operation of a robot, the method comprising: obtainingmeasurement information of a processing target that includes at leastone of the robot and a target that the robot operates; generating, basedon the measurement information, display information that is viewable bya viewer; generating emphasized display information that emphasizes afeature of a portion of the processing target by magnifying or changingangle of or deforming the processing target, based on the displayinformation; generating composite display information in which theemphasized display information and non-emphasized display informationother than the emphasized display information are composed, thecomposite display information being generated by matching the emphasizeddisplay information and the non-emphasized display information in thedisplay information; and outputting the composite display information toa display device.
 17. A non-transitory computer-readable storage mediumstoring a computer program for causing a computer to execute a displaycontrol method that displays an operation of a robot, the methodcomprising: obtaining measurement information of a processing targetthat includes at least one of the robot and a target that the robotoperates; generating, based on the measurement information, displayinformation that is viewable by a viewer; generating emphasized displayinformation that emphasizes a feature of a portion of the processingtarget by magnifying or changing angle of or deforming the processingtarget, based on the display information; generating composite displayinformation in which the emphasized display information andnon-emphasized display information other than the emphasized displayinformation are composed, the composite display information beinggenerated by matching the emphasized display information and thenon-emphasized display information in the display information; andoutputting the composite display information to a display device.